-
Notifications
You must be signed in to change notification settings - Fork 3.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
HBASE-27149 Server should close scanner if client times out before results are ready #4604
Conversation
🎊 +1 overall
This message was automatically generated. |
🎊 +1 overall
This message was automatically generated. |
🎊 +1 overall
This message was automatically generated. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
lgtm
…sults are ready (#4604) Signed-off-by: Andrew Purtell <apurtell@apache.org>
…sults are ready (#4604) Signed-off-by: Andrew Purtell <apurtell@apache.org>
…imes out before results are ready (apache#4604) Signed-off-by: Andrew Purtell <apurtell@apache.org>
I have a question about the sentence
in issue HBASE-27149. Could anyone explain the evidence that shows the server is slowed down? I am trying to reproduce this error, but I find that the scanning time after the timeout scans still does not change. |
This issue has been fixed for about 6 months, so I don't have any evidence to hand anymore. You won't be able to reproduce it if you have the patch. Basically you can see that the scanner holds resources on the server side prior to this patch, even if the client times out. Ideally I'd the client times out it will try to close the scanner with another RPC. But if the server is overloaded that close RPC won't even make it to the server. In this case the scan lives on. It's not actively scanning data but just holding memory. There is an activeScanners metric you can inspect in these cases. Anyway no I can't provide any hard data at this point but hopefully that should give you some pointers for what to look for, as long as you don't have this patch. It's also possible to mitigate this by setting the server side lease period low. But this is hard to do in a multi tenant environment where one timeout does not fit all cases. |
In terms of how to tell if server is slowed down enough. Look at queue times and response times, which might both be in the seconds. You'd expect to see a similar saturation in cpu and/or disk metrics. |
Thank you for your quick reply!
Yes, I find that if I set
OK, I will try it by running on a slow container and disabling the block cache (since the scanning is fast, I have to make it slow for a better observation), and then check the response time. Thank you for your guidance. |
No description provided.